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Abstract 

Background: Foot disease complications, such as foot ulcers and infection, contribute to considerable morbidity 
and mortality. These complications are typically precipitated by "high-risk factors", such as peripheral neuropathy 
and peripheral arterial disease. High-risk factors are more prevalent in specific "at risk" populations such as diabetes, 
kidney disease and cardiovascular disease. To the best of the authors' knowledge a tool capturing multiple high-risk 
factors and foot disease complications in multiple at risk populations has yet to be tested. This study aimed to 
develop and test the validity and reliability of a Queensland High Risk Foot Form (QHRFF) tool. 

Methods: The study was conducted in two phases. Phase one developed a QHRFF using an existing diabetes foot 
disease tool, literature searches, stakeholder groups and expert panel. Phase two tested the QHRFF for validity and 
reliability. Four clinicians, representing different levels of expertise, were recruited to test validity and reliability. 
Three cohorts of patients were recruited; one tested criterion measure reliability (n = 32), another tested criterion 
validity and inter-rater reliability (n =43), and another tested intra-rater reliability (n = 19). Validity was determined 
using sensitivity, specificity and positive predictive values (PPV). Reliability was determined using Kappa, weighted 
Kappa and intra-class correlation (ICC) statistics. 

Results: A QHRFF tool containing 46 items across seven domains was developed. Criterion measure reliability of at 
least moderate categories of agreement (Kappa > 0.4; ICC > 0.75) was seen in 91% (29 of 32) tested items. Criterion 
validity of at least moderate categories (PPV> 0.7) was seen in 83% (60 of 72) tested items. Inter- and intra-rater 
reliability of at least moderate categories (Kappa > 0.4; ICC > 0.75) was seen in 88% (84 of 96) and 87% (20 of 23) 
tested items respectively. 

Conclusions: The QHRFF had acceptable validity and reliability across the majority of items; particularly items 
identifying relevant co-morbidities, high-risk factors and foot disease complications. Recommendations have been 
made to improve or remove identified weaker items for future QHRFF versions. Overall, the QHRFF possesses 
suitable practicality, validity and reliability to assess and capture relevant foot disease items across multiple at risk 
populations. 
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Background 

Foot disease contributes to considerable hospitalisation 
[1-4], amputation [5-8], institutionalisation [9,10], and 
death [2,11]; yet foot disease complications and these 
outcomes are largely preventable [5,6,12,13]. "High risk 
factors" (such as peripheral neuropathy, peripheral arter- 
ial disease or foot deformity) significantly increase the 
risk of developing "foot disease complications" (such as 
foot ulcers, infection or ischaemia) [1,14-16]. Diabetes 
populations are frequently acknowledged as the leading 
"at risk population" for foot disease due to the increased 
risk of developing high risk factors, and in turn foot 
disease complications, from diabetes [5-8,17]. However, 
other chronic disease populations (such as chronic kid- 
ney disease [15,16], cardiovascular disease [5,6,18,19] and 
some cancers [5,6,8]) have now been shown to cause com- 
parable rates of high risk factors and foot disease compli- 
cations to diabetes, and thus, are also becoming more 
readily identified as "at risk populations" for foot disease. 

Best practice foot disease management has been shown 
to significantly reduce hospitalisation, amputation, mortal- 
ity and overall costs within different at risk populations 
[5,12,13,20]. These multi-faceted best practice interven- 
tions commonly include screening for high-risk factors, 
multi-disciplinary management of foot disease complica- 
tions, clinical training, evidence-based clinical pathway 
utilisation and regular capture and analysis of foot disease 
clinical data [5,6,12,13,20]. In consideration of the growing 
problem of foot disease, and the potential future improve- 
ments observed with best practice clinical management 
and research, it is imperative that any clinical tools to as- 
sess, capture, measure or analyse patient outcomes in at 
risk populations are valid and reliable. 

There are a multitude of studies that have investigated 
a specific high risk foot factor or foot disease complication 
within multiple at risk populations [3,5,6,21]. Furthermore, 
numerous studies have investigated multiple high risk 
factors and foot disease complications in specific at risk 
populations such as diabetes [2,15,17]. However, very few 
studies have investigated multiple high risk foot factors 
and foot disease complications in multiple at risk popula- 
tions. This situation appears to be mirrored by the clinical 
tools available to measure and report on foot disease. For 
example many validated single-item tools are available 
which measure specific high-risk factors or foot disease 
complications within multiple at risk populations [22-24]; 
such as the ankle brachial index to capture and measure 
peripheral arterial disease within various at risk popula- 
tions [22]. Furthermore, many validated multi-item tools 
measuring multiple high-risk factors and foot disease 
complications in specific at risk populations have been de- 
veloped [15,25-28]; such as the University of Texas 
Diabetic Foot Classification System to capture and meas- 
ure multiple high risk factors and foot disease 



complications in people with diabetes only [25]. However, 
to the best of the authors' knowledge a multi-item tool de- 
signed to measure multiple high-risk factors and foot dis- 
ease complications in multiple at risk populations is yet to 
be developed and tested for validity and reliability. 

Recently a Diabetic Foot Form (DFF) tool was developed 
to measure multiple high-risk factors and foot disease 
complications in people with diabetes only in diverse 
Queensland (Australia) settings [29] . The DFF was a multi- 
item tool developed from a number of existing clinical 
tools recommended in the literature to reliably capture 
various high-risk factors and foot disease complications 
in the diabetes population [29]. The implementation of 
the DFF in diverse Queensland clinical sites, in conjunc- 
tion with other multi-faceted strategies, resulted in im- 
proved capture, measurement and management of high 
risk factors and foot disease complications and a corre- 
sponding reduction in outcomes such as hospitalisation 
and amputation [29,30]. However, the tool was not tested 
for validity or reliability and was designed only to measure 
multiple high-risk factors and foot disease complications 
in the specific at risk population of people with diabetes. 

In this study, we sought to modify the DFF tool to en- 
able the measurement of multiple high risk factors and 
foot disease complications in multiple at risk populations. 
It was intended that the new multi-item tool would align 
with best practice principles for clinical tools including be- 
ing easily interpreted, practical to use and possessing high 
face, content and criterion validity, and inter- and intra- 
rater reliability [31-33]. Thus, the aims of this study were 
firstly to develop a multi-item Queensland High Risk Foot 
Form (QHRFF) tool to capture multiple high risk factors 
and foot disease complications in multiple applicable at 
risk populations, and secondly, to investigate the validity 
and reliability of the tool's individual items when used by 
clinicians representing different levels of foot disease 
expertise. 

Methods 

The study was conducted in two phases. Phase one in- 
volved development of the Queensland High Risk Foot 
Form (QHRFF) tool and phase two tested the validity and 
reliability of the QHRFF tool. Approval was granted from 
Institutional Ethics Committees and informed consent 
was obtained from all individual participants (patients and 
clinicians) for this study. 

Phase one - development of the tool 

Phase one primarily aimed to select items for the develop- 
ment of a QHRFF tool that were practical to collect in an 
Australian clinical setting, applicable to multiple 'at risk' 
populations, and provided high face and content validity. 
The overarching procedures used for item selection in- 
cluded using the original DFF [29] as the starting tool to 
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build the new QHRFF, searching the electronic literature 
for other recommended foot disease tools, establishing an 
expert panel to guide development and using several 
rounds of stakeholder consultation to refine the tool. 

The original DFF contained 64 items pertaining to the 
construct of foot disease in diabetes populations [29] 
and was used as the starting tool to modify into the 
QHRFF. The DFF was originally developed using similar 
procedures to those used in this study and appeared to 
possess high practicality, face and content validity [29]. 
Furthermore, the DFF had been routinely used in over 
25 High Risk Foot Service sites throughout Queensland 
for three years to collect standard clinical data on diabetes- 
related foot disease [30] . 

An electronic literature search was undertaken of rele- 
vant electronic databases including MEDLINE (all years to 
June 2011), CINAHL (all years to June 2011) and relevant 
professional websites to identify existing foot disease- 
related tools. The basic terms searched included tools to 
identify peripheral neuropathy, peripheral arterial disease, 
foot deformity, foot ulceration, foot infection and amputa- 
tion or synonyms. 

An expert panel was established from members of the 
Queensland Statewide Diabetes Foot Working Group; a 
working group of the Queensland Statewide Diabetes 
Clinical Network. The panel comprised 14 expert clini- 
cians, with between five and 25 years of experience in 
the area of foot disease management, from the fields of 
endocrinology, vascular surgery, podiatry, nursing, pub- 
lic health, quality improvement and research. The panel 
had the responsibility to decide upon the final items for 
the QHRFF tool after each round of consultation. 

Refinements of the tool were achieved through numerous 
rounds of consultation and trialling of draft QHRFF ver- 
sions with external stakeholders in relation to practicality, 
face and content validity. Stakeholders included up to 200 
multidisciplinary professionals from sites registered to use 
the existing DFF, members of the Queensland Health 
Statewide Diabetes Clinical Network, Statewide Renal 
Clinical Network and Statewide Podiatry Network. 

At the conclusion of phase one the expert panel and 
aforementioned network's management committees deter- 
mined, via consensus endorsement, that each item of the 
QHRFF possessed high practicality, face and content valid- 
ity, and was applicable across multiple 'at risk' populations 
(particularly diabetes, cardiovascular disease and chronic 
kidney disease) in Australian clinical settings. See Phase 
One Results for further details regarding the final QHRFF. 

Phase two - validity and reliability 

The final endorsed QHRFF tool developed in phase one 
was then tested for validity and reliability. The general 
procedure for testing involved using podiatrists with dif- 
ferent levels of foot disease expertise to assess patients 



with different levels and severity of high risk factors and 
foot disease. The authors decided to test only QHRFF 
items that directly related to the foot disease construct, 
thus, excluding items such as patients' name, and facility. 
To test the validity of each item a general criterion meas- 
ure (the agreement between two 'experts') was initially 
assessed for reliability. Each item was then tested, using 
the podiatrists with different experience levels, against the 
criterion measure to evaluate magnitudes of concurrent 
criterion validity. The inter- and intra-rater reliability was 
also tested for the level of agreement on each item by po- 
diatrists with different levels of experience. 

Setting and participants 

The testing was conducted within the High Risk Foot 
Services (HRFS), Metro North Hospital and Health 
Service, Brisbane, Australia. Eligible clinician participants 
were recruited from podiatrists practicing a minimum of 
one session per week in a HRFS. Four podiatrists were 
chosen as representative of the majority of clinicians man- 
aging foot disease within Queensland Health, and thus, 
potentially using the QHRFF in future. For the purposes 
of the study, levels of expertise were categorised using the 
Queensland Health 'Health Practitioner award' [34]. Thus, 
expert clinicians were defined as either a 'consultant clin- 
ician' (level 6) or 'specialist clinician' (level 5) [34] and 
working in an acute hospital setting, plus, a member of 
the expert panel to ensure they understood the original 
intended construct for the QHRFF tool. General clinicians 
were defined as a 'senior clinician' (level 4) or 'clinician' 
(level 3) [34], working in a community setting, and 
thought to be representative of the general podiatry clin- 
ical workforce managing foot disease in Queensland. One 
of each level was recruited; one level 6, one level 5, one 
level 4 and one level 3 (however, the level 3 had been act- 
ing in a level 4 position at the time of the study). Written 
informed consent was obtained from all participants 
(patients and clinicians) prior to commencement of their 
study participation. 

Eligible patient participants were consecutively recruited 
from patients already attending Community HRFS clinics 
for the care of high risk factors or foot disease complica- 
tions; defined as a previous or current foot ulcer. Exclu- 
sion criteria included patients with a cognitive deficit, 
signs or symptoms of a systemic infection, younger than 
18 years of age, or unwilling to provide written consent to 
participate. The authors considered that patients with pre- 
vious or current foot ulcers would ensure that the majority 
of high risk factors and foot disease items had the realistic 
possibility of being present or absent, and thus, could be 
suitably tested on each participant. Furthermore, it was 
thought this population should possess the moderate 
prevalence rates, yet unpredictable mixes and severities of 
each item, of different high risk factors and foot disease 
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complications, that are suggested in the literature to im- 
prove statistical robustness for validity and reliability 
studies [31]. 

Three different patient cohorts were used; one cohort to 
test the reliability of the criterion measures (agreement be- 
tween experts') (n = 32), another cohort to simultaneously 
test the criterion validity (an 'expert' diagnosis against 



general clinicians) and inter-rater reliability (n = 43), and 
the last cohort to test the intra-rater reliability of a general 
clinician (n = 19). The recruitment of consecutive commu- 
nity patients did mean that patients may have been famil- 
iar to the level 3 or level 4 clinicians; however, not to the 
expert 'gold standard' clinicians working in the hospital 
setting. To minimise the risk of patients being familiar to 
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Date of birth: 
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High Risk Foot Data Collection Form 



Facility: 



(Reese enter your 
Facility name or code) 



Visit type O New client visit 
(single choice) 

Today's visit to HRFS 



O Review client visit C Did not attend 



Separation status (If appicable) 
O Discharged O Transferred 
(single choice) 



Deceased 



Health professionals) 
attending (multiple choice) 
O Podiatrist O GP 
O Nurse O Orthotist 
O Physician O Surgeon 
O Other 



New client visits only (or client previously discharged with new referral) 



Date of referral 



Indigenous status O Aboriginal O Torres Strait Islander O Both 



(single cnotce) 



O Neither O Unknown/ Not stated 



SUBJECTIVE Reason for attendance 



Medical and diabetic foot history 



Medical history o Non-diabetes 
(single choice) 



O Type 1 diabetes O Type 2 diabetes 



If diabetes, then 
year diagnosed: 



Co- morbidities 0 Neuropathy 
(multiple choice) Q PAD 

O Other: 



O Hypertension 
O Smoker 



O Dyslipidaemia 
OCKD 



O CVD O ESRF 



Recent BGLs greater than 15mmol/L? 

OYes ONo ON/A 



HbA1c result 



□ 



Previous foot ulcer O Yes O No 
Current foot ulcer OYes ONo 

I — ► If yes, Is this a new ulcer? 
Previous amputation o Yes O No 



) Yes O No 



Clinical diagnosis 



Neuropathy O Yes □ No 

PAD O Nil O Mod O Crit Location:. 

Acute Charcot O Yes O No Location:. 

Foot Deformity O Yes O No 



Ulcer / wound assessment summary 



Type (single choice) 

(for highest scoring ulcer / wound) 

O Neuropathic O Neuro-lschaemic 
O Ischaemic O Post Surgical 

Combined surface area 



Risk Classification (single choice 
Acute 



High risk 
At risk 
Low risk 



- Foot ulcer/ acute Charcot 

- Foot deforrrity wtth Neuropathy and / or PAD 

- Previous ulcer or amputation or critical PAD 

- Neuropathy or PAD 

- Nil Neuropathy or PAD 



Combined surface area (single choice] 

O Healed O No change O Smaller O Larger 

Clinical signs of infection (see infection - page 4) 
O Nil O Mild O Moderate O Systemic 



UTWCS Grade 



(for highest scoring ulcer/ wound) 





Debrided ulcer / wound / callus O Yes ONo 


ON/A 


Off-loading optimum 


OYes ONo 


O N/A 


Dressing optimum 


O Yes O No 


ON/A 


Footwear optimum 


OYes ONo 


O N/A 


Antibiotics required 


O Yes O No 


O N/A 


Educated patient 


OYes ONo 


ON/A 


Completed by (print name) 


Designation: 


Signature: 


Data: 


Photocopy this form, all 


n photocopy and retain for chart. Sens 
Pano2oM 


original to data collections officer. PSOJS 

1212594981 



Figure 1 Queensland high risk foot data collection form (QHRFF). 
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the level 3 or level 4 clinicians' seven different Community 
HRFS clinics were used to recruit patient participants. 
The level 3 and level 4 clinicians had only worked at two 
of the seven clinics recruiting patients. 

Procedures 

A designated research assistant coordinated all proce- 
dures. Training of each clinician consisted of being pro- 
vided with a QHRFF manual that gave a definition of 
each item [35], a 1-2 hour training session on instruc- 
tions and tips to use the QHRFF tool, and each was en- 
couraged to trial the tool on their existing patients and 
clarify any queries with the research assistant prior to 
testing. 

The general assessment procedure for each validity or 
reliability test consisted of patients having their feet 
examined by at least two different clinicians, blinded from 
each other's assessment, within the one clinical visit (valid- 
ity and inter-rater reliability). However, to ensure all clini- 
cians had an equal opportunity to determine the patient's 
debridement and wound management needs, all were per- 
mitted to visually inspect the patient's feet together for up 
to five minutes prior to any clinician ratings. In this initial 
inspection period, the clinicians were instructed to only 
visually inspect the need for debridement and previous 
wound dressings whilst not conversing with or touching 
the patient or each other. The order of clinician assess- 
ment after this inspection period was then at the dis- 
cretion of the research assistant based on clinician 
availability. The first clinician would have the additional 
task of debriding the wound or callus if they deemed 
necessary and the last clinician the additional task of 
completing any clinical management. 

The research assistant ensured all clinicians were 
blinded from each other's assessments in separate rooms 
and that all examination records were de-identified. 
Each clinician conducted the assessment of patient's feet 
using the QHRFF to record their assessment and man- 
agement recommendations. Demographic, medical history 
and medication information were available from the pa- 
tients' medical records or by direct communication with 
the patients themselves. To minimise assessment bias all 
historical foot-related records (including progress notes, 
DFF tools or QHRFFs tools) were removed from the med- 
ical record prior to each clinician's assessment. 

Criterion measure The criterion measure was tested on 
the first cohort of patients. The criterion measure (i.e. 
'gold standard' diagnosis) for each QHRFF item in this 
study was defined by the authors as the diagnosis made 
by an expert clinician. Expert clinicians were pragmatic- 
ally chosen as a general criterion measure for all items, 
instead of using multiple resource and time intensive 
individual gold standard-recognised criterion measures 



(such as nerve conduction studies for neuropathy), due 
to the resource and time constraints of the study. Best 
practice dictates that any criterion measure should be 
reliable, free from bias and measure the same item as 
the new tool [31]. Thus, the study's criterion measure, of 
an expert clinician, required testing to determine its reli- 
ability, independence (free from bias) and applicability, 
prior to its use as a 'gold standard' criterion measure in 
the criterion-related validity tests. The authors tested the 
criterion measure by testing the reliability of the agree- 
ment between two independent (blinded from one an- 
other) experts (one level 6 and one level 5) to determine 
the magnitude of reliability of agreement on their 'gold 
standard' diagnoses for each item on the same patient 
cohort. The "general assessment procedure" as described 

Table 1 Supporting tools or literature for QHRFF items 

QHRFF item Supporting tool or literature 

General demographics* [35] 

Indigenous status [14] 

Health professionals attending* [14,39] 

Medical co-morbidity history* [40,41] 

Medical (diabetes) history [5-8] 

Diabetes year diagnosed (duration) [1 7,42] 

Recent BGLs > 1 5 mmol/L [41 ,42] 

HbA 1c result [41,42] 

CKD [41,43] 

ESRF [41,43] 

High-risk factor history* [14,35,39] 

Clinical diagnoses of high-risk factors 

Neuropathy Monofilament test 

[1,14,23,39,44,45] 



PAD 

Acute Charcot 
Foot deformity 



Risk classification 
Clinical diagnoses of foot disease 

(Ulcer) type [14,53,54] 

Combined surface area mm 2 [54-56] 

Combined surface area (change since [55-59] 
last visit) 



Foot pulses, ankle brachial index 
and/or toe systolic pressure 
[1,14,21,22,39,46-49] 

[14,39] 

Six-point foot deformity scale 
[14,50] 

[14,39,51,52] 



Clinical signs of infection 
UTWCS grade 
Ulcer depth* 

Clinical management principles 
performed* 



[1,54,60,61] 

UTVJCS tool [14,62,63] 

[1,14,54] 

[14,39] 



*AII items in this Domain cite the same references, unless otherwise stated. 
#Ulcer depth was not specifically an item recorded on the QHRFF, but can be 
directly extrapolated from the UTWCS Grade item. 
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Table 2 Demographic, co-morbidity and high-risk foot complication data for each patient cohort (n (%) unless 
otherwise stated) 



Criterion measure 



Criterion validity* 
& inter-rater reliability* 



Intra-rater reliability 



p Value 



General demographics 

Numbers 32 

Male 25 (78%) 

Age, mean (SD) (years)# 69(13) 

Age range (years) 48 - 90 

ndigenous 1 (3%) 
Health professionals attending (in the past week) 

Podiatrist 32(100%) 

GP 2 (6%) 

Nurse 3 (9%) 
Orthotist 0 

Physician 2 (6%) 

Surgeon 1 (3%) 
Other 0 
Medical co-morbidity history 

Diabetes (type 1 or 2) 28 (88%) 

Type 2 diabetes 28 (88%) 
Diabetes duration (years)# 12(7) 

Recent BGLs > 1 5 mmol/L 5 (1 6%) 

HbA1c(SD)# 9.2(3.5) 

Hypertension 28 (88%) 

Dyslipidaemia 23 (72%) 

Smoker 5(16%) 

CVD 21 (66%) 

CKD 5(16%) 
ESRF 0 

Other (non-listed) 29(91%) 
High-risk history 

Neuropathy 31 (97%) 

PAD 14(44%) 

Previous foot ulcer 27 (84%) 

Current foot ulcer 32 (100%) 

New foot ulcer 21 (66%) 

Previous amputation 16(50%) 
Clinical diagnoses of high-risk factors 

Neuropathy 31 (97%) 

Any PAD 14(44%) 

Acute Charcot 1 (3%) 

Foot deformity 26(81%) 

Acute risk classification 32 (100%) 



43 

37 (86%) 
68 (13) 
36 - 89 

0 (0%) 

43 (100%) 
10 (23%) 
1 2 (28%) 
0 

1 (2%) 
3 (7%) 

0 



35 (81%) 
31 (72%) 

23 (11) 
4 (9%) 
8.1 (1.6) 
30 (70%; 
25 (58%; 
11 (26%; 
21 (49%: 
1 3 (30%: 
1 (2%) 

36 (84%: 

37 (86%: 
21 (49%: 
37 (86%: 
42 (9; 
42 (98%: 
1 7 (40%; 



22 (51% 
1 (2%) 
33 (77%; 
42 (98%; 



19 

1 7 (90%) 
70 (15) 
42 - 90 

1 (5%) 

19 (100%) 

2 (11%) 

4 (21%) 

0 

1 (5%) 
0 
0 

16 (84%) 
14 (74%) 
20 (14) 

5 (26%) 
8.4 (2.7) 
11 (58%) 
9 (47%) 

3 (16%) 
9 (47%) 
5 (26%) 

0 

11 (58%) 

1 8 (95%) 
8 (42%) 

1 7 (90%) 

19 (100%) 
19 (100%) 

7 (37%) 

19 (100%) 

8 (42%) 

0 

1 7 (90%) 
19 (100%) 



0.501 
0.826 

NA 

1.000 
0.103 
0.141 

NA 

NA 

NA 

NA 

0.384 
0.256 

0.001** 
0.220 
0.450 
0.053 
0.201 
0.492 
0.281 
0.339 
NA 

0.013** 

0.214 
0.853 
0.878 
0.549 
< 0.001* 
0.566 

0.146 
0.735 
NA 
0.501 
0.549 
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Table 2 Demographic, co-morbidity and high-risk foot complication data for each patient cohort (n (%) unless 
otherwise stated) (Continued) 



Clinical diagnoses of foot disease 










Ulcer type: neuropathic 


14 (44%) 


14 (33%) 


8 (42%) 


0.572 


Ulcer type: neuroischaemic 


9 (28%) 


16 (37%) 


8 (42%) 


0.555 


Ulcer type: ischaemic 


0 


3 (7%) 


1 (5%) 


NA 


Ulcer type: other 


9 (28%) 


9 (21%) 


2 (11%) 


0.339 


Combined surface area (mm 2 )# 


122 (191) 


352 (899) 


379 (1436) 


0.469 


Any clinical infection 


6 (19%) 


14 (33%) 


2 (11%) 


0.125 


UTWCS (Grade 1A) 


13 (41%) 


14 (33%) 


7 (37%) 


0.770 


Deep ulcer (Grade 2 or 3) A 


8 (25%) 


6 (14%) 


2 (11%) 


0.394 



* = Same participants used for three tests. 
** = Statistically significant (p < 0.05). 

# = Measure in mean (standard deviation). 

a = Deep ulcer determined from the UTWCS Grade item. 

NA = Not applicable to test as the assumption of Chi-squared test is violated as 2 cells had expected counts of < 5. 

GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; PAD = Peripheral Arterial Disease; UTWCS = University of Texas Wound 
Classification System. 



above was utilised for each patient. Once the criterion 
measure for each item was determined to be reliable, the 
criterion validity of each QHRFF item was tested. 

Criterion validity The criterion validity and inter-rated 
reliability was then tested simultaneously on a second 
cohort of patients. Criterion validity was tested using the 
concurrent criterion validity method by comparing the 
reliable criterion measure (one of the expert clinicians) 
against each of the two representative 'general' clinicians 
(one level 3 and one level 4). Thus, two separate criter- 
ion assessments were carried out for each item; one 
using the level 3 clinician against the criterion measure, 
and another the level 4 clinician against the criterion 
measure. Again the "general assessment procedure" as 
described above was utilised for each patient when test- 
ing for criterion validity. 

Inter-rater reliability Inter-rater reliability tests were 
performed simultaneously with the criterion validity tests 
on the same second cohort of patients and were tested 
using the expert clinician (level 5), senior clinician 
(level 4) and clinician (level 3). Thus, three inter-rater reli- 
ability measures of agreements were carried out for each 
item; one testing the agreement between the level 5 and 
level 4 clinician, a second between the level 5 and level 3 
clinician, and a third between the level 4 and level 3 clin- 
ician. "General assessment procedures" as described above 
were again utilised for each patient when testing for inter- 
rater reliability. 

Intra-rater reliability Intra-rater reliability was tested 
on a third cohort of patients. The clinician with the least 
expertise (level 3) was used to test intra-rater reliability 
as it was hypothesised that the clinician with the least 



expertise would have the most variability of the tested 
clinicians. The two different time points to test intra- 
rater reliability were between one and four weeks apart. 
This time period was chosen as it was considered that a 
minimum of one week (of full clinical load) would be 
necessary to adequately reduce recall bias in a clinician, 
plus, a maximum of four weeks would not be sufficient 
time for the majority of items to markedly change and 
thus items would remain stable between ratings. Any 
items that did not fit this criteria were excluded from 
intra-rater testing. Furthermore, this time period aligned 
with any necessary follow up treatment time period for 
patients, and thus, was convenient to patient partici- 
pants. To control for any potential changes in foot ulcer 
characteristics over time (for example ulcer combined 
surface area and clinical signs of infection) digital images 
were used. Digital photographic images taken of foot 
ulcers at the first rating were used at the second rating 
to standardise the foot ulcer characteristics across two 
time points [36]. Images were taken perpendicular and 
30-60 cm away from ulcers at the first rating, and incor- 
porated two calibrated measures across the length and 
width of the ulcer in the image. These images were 
reviewed by the clinician at the second rating to deter- 
mine the foot ulcer characteristics only. 

Statistical analysis 

All data was analysed using SPSS 19.0 for Windows 
(SPSS Inc., Chicago, IL, USA) or GraphPad Software. De- 
scriptive statistics were used to display the single demo- 
graphic, medical co-morbidity, high-risk factors and foot 
disease variables for each cohort of patient participants; 
using means and standard deviations (SD) for continuous 
variables (which were normally distributed) or proportions 
for categorical variables. Each above descriptive cohort 
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Table 3 Criterion measure reliability - measure of agreement between gold standard experts 



QHRFF item 



% 



MSE) 



Strength of agreement [37] 



77.4 

) 

96.9 
81.2 
84.4 
100 
96.9 
100 
100 

100 



General demographics 

ndigenous status* 

Health professionals attending (in the past week) 

Podiatrist 
GP 

Nurse 
Orthotist 
Physician 
Surgeon 
Other 

Medical co-morbidity history 

Medical (diabetes) history 
Diabetes duration* 
Recent BGLs > 15 mmol/L* 
HbA1c result* 
Hypertension 
Dyslipidaemia 
Smoker 
CVD 
CKD 
ESRF 

Other (non-listed) 
High-risk history 

Neuropathy 
PAD 

Previous foot ulcer 
Current foot ulcer 
New foot ulcer 
Previous amputation 
Clinical diagnoses of high-risk factors 
Neuropathy 
PAD* 

Acute Charcot 
Foot deformity 
Risk classification* 

Clinical diagnoses of foot disease 

Ulcer type* 
Combined surface area mm 2 # 

Clinical signs of infection* 84.3 
UTWCS grade* 

Ulcer depth* 78.1 



87.5 
75.0 
96.9 
93.8 
100 
96.9 
46.9 

93.8 
84.4 
84.4 
96.9 
83.9 
90.6 

96.9 
77.4 
100 
81.3 



87.5 



0.45 (0.17) 



0.78 (0.21) 
0.47 (0.18) 

0.78 (0.21) 
1.00 (0.01) 



1.00 (0.01) 
0.94 (0.88-0.? 
0.81 (0.11) 
0.93 (0.79-0.? 
0.60 (0.17) 
0.46 (0.16) 
0.89 (0.11) 
0.86 (0.10) 
1.00 (0.01) 

0.04 (0.09) 

0.48 (0.31) 
0.68 (0.13) 
0.45 (0.21) 

0.58 (0.15) 
0.81 (0.10) 

0.65 (0.32) 
0.61 (0.12) 
1.00 (0.01) 
0.46 (0.19) 



0.85 (0.08) 
0.73 (0.52-0.86) 
0.68 (0.15) 
0.56 (0.28-0.76) 
0.67 (0.13) 



Moderate 



Substantial 
Moderate 

Substantial 
Near perfect 



Near perfect 
Strong 
Near perfect 
Strong 
Moderate 
Moderate 
Near perfect 
Near perfect 
Near perfect 

Slight 

Moderate 

Substantial 

Moderate 

Moderate 
Near perfect 

Substantial 
Substantial 
Near perfect 
Moderate 



Near perfect 

Moderate 

Substantial 

Weak-moderate 

Substantial 
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Table 3 Criterion measure reliability - measure of agreement between gold standard experts (Continued) 

Clinical management principles performed 

Debrided ulcer/callus* 93.8 

Dressing optimum* 96.9 

Antibiotics required* 84.4 

Off-loading optimum* 68.8 

Footwear optimum* 65.6 

Educated patient* 100 

K = Kappa; (SE) = Standard Error; % = Percentage agreement. 
- = Measure of agreement was unable to be calculated because at least one variable was a constant. 

GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; PAD = Peripheral Arterial Disease; UTWCS = University of Texas Wound 
Classification System. 

* = Weighted Kappa used for ordinal data. 

# = Intraclass correlation (ICC) (Standard Deviation (SD)} for continuous data. 
A = Depth was determined from the UTWCS score. 



0.47 (0.19) 
0.41 (0.14) 
0.28 (0.16) 



Moderate 
Moderate 
Fair 



variable was collected from the clinician with most expert- 
ise or on the first rating of the intra-rater reliability cohort. 
Chi-squared test of independence and ANOVA were 
used to test for differences between the three cohorts' 
characteristics. A significance level of p < 0.05 was used 
throughout. 

All QHRFF items were tested for validity and reliability; 
except those stated items considered not to impact on the 
foot disease construct. Criterion validity was tested by cal- 
culating sensitivity, specificity and positive predictive 
values. Ordinal variables were collapsed into dichotomous 
data to enable calculations of sensitivity, specificity and 
positive predictive values. All measures of agreements 
between clinicians were tested using Kappa (K) for di- 
chotomous variables, weighted Kappa (wK) for ordinal 
variables and intra-class correlation (ICCs) (model 2, 1) 
for continuous variables [31,37,38]. Kappa and wK value 
(Standard Errors (SE)) strengths were categorised as: no 
agreement < 0; slight agreement = 0 - 0.20; fair agree- 
ment = 0.21 - 0.40; moderate agreement = 0.41 - 0.6; 
substantial agreement = 0.61 - 0.8; and near perfect 
agreement = 0.81 - 1.0 [31,37,38]. ICC (SD) strengths 
were categorised as: weak-moderate agreement < 0.75 and 
strong agreement > 0.75 [31]. 

Results 

Phase one - development of the tool 

The literature search identified 174 papers that reported 
on tools to identify or measure foot disease complications 
and/or high-risk factors, with the majority of papers spe- 
cifically relating to diabetes populations (90 (52%)). 
Tools reported in other populations were in conditions 
commonly identified in the literature to be associated 
with lower limb amputation [5-8]. These included car- 
diovascular disease (peripheral arterial disease), chronic 
kidney disease, malignancy, infection and other neuro- 
logical conditions [5-8]. No tool was identified that was 
specifically designed to identify multiple high-risk 



factors and foot disease complications in multiple at 
risk populations. 

Fifty-nine individual tools were identified from the lit- 
erature search; 23 were considered to have adequate 
practicality, face validity and applicability to an Australian 
clinical context to be considered for the QHRFF. These 
tools and the original DFF tool [29] were considered by 
the expert panel for QHRFF item selection. At this 
point, the expert panel determined that the tool should 
be divided into a clinical assessment record section that 
informed a separate data collection section, and thus, 
only the data collection section would require testing. 

Overall 87 items were initially identified from the con- 
sidered tools. At the conclusion of phase one procedures 
the final endorsed QHRFF data collection tool was made 
up of 46-items (excluding general identification items) 
covering seven domains (Figure 1). The seven content 
domains included identifying general demographics, differ- 
ent health professionals attending, medical co-morbidity 
history, high-risk factor history, clinical diagnosis of high- 
risk factors, clinical diagnoses of foot disease complica- 
tions, and clinical management principles performed. A 
QHRFF manual was developed to provide definitions and 
instructions on each item contained in the tool [35]. 
Table 1 outlines the tools or literature used to support 
each QHRFF item. At the conclusion of phase one, the ex- 
pert panel concluded that the QHRFF tool's items had the 
required high clinical practicality, applicability to multiple 
at risk populations, and good face and content validity. 
Subsequently, the tool was endorsed for use by the 
Queensland Health Statewide Diabetes, Renal and 
Podiatry Clinical Networks. Thus, the tool was ready for 
validity and reliability testing. 

Phase two - validity and reliability 

Forty items were tested for validity and reliability un- 
less otherwise stated. The items not tested were those 
considered not to impact on the foot disease construct; 
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Table 4 Criterion validity summary statistics for senior clinician (Sensitivity, specificity and positive predictive value) & 
inter-rater reliability between expert and senior clinicians (Kappa and strength of agreement) 

QHRFF field % K (SE) Strength of Sensitivity Specificity (95% CI) PPV (95% CI) 

agreement [37] (95% CI) 

General demographics 

Any indigenous status 100 - - - 1.00(0.90-1.00) 

Health professionals attending (in the past week) 



Podiatrist 


100 










1.00 (0.90-1.00) 




1.00 (0.90-1.00) 


GP 


86.0 


0.64 


(0.14) 




Substantial 


0.80 (0.44-0.96) 


0.88 (0.71-0.96) 


0.67 (0.35-0.88) 


Nurse 


93.0 


0.83 


(0.09) 




Near perfect 


0.92 (0.60-1.00) 


0.94 (0.77-0.99) 


0.85 (0.54-0.97) 


Orthotist 


100 












1.00 (0.90-1.00) 




Physician 


93.0 










0 (0 - 0.95) 


0.95 (0.83-0.99) 


0 (0 - 0.80) 


Surgeon 


95.3 


0.64 


(0.24) 




Substantial 


0.67 (0.13-0.98) 


0.98 (0.85-1.00) 


0.67 (0.13-0.98) 


Other 


100 












1.00 (0.90-1.00) 




Medical co morbidity history 


















Any medical (diabetes) history 


100 


1.00 


(0.01) 




Near perfect 


1.00 (0.88-1.00) 


1.00 (0.60-1.00) 


1.00 (0.88-1.00) 


Diabetes duration* 




0.90 


(0.80- 


0.95) 


Strong 








Recent BGLs > 15 mmol/L 


93.0 


0.69 


(0.16) 




Substantial 


0.57 (0.20-0.88) 


1.00 (0.88-1.00) 


1.00 (0.40-1.00) 


HbA1c results 




1.00 


(0.99- 


1.00) 


Strong 








Hypertension 


93.0 


0.83 


(0.09) 




Near perfect 


0.97 (0.81-1.00) 


0.85 (0.54-0.97) 


0.94 (0.77-0.99) 


Dyslipidaemia 


95.3 


0.90 


(0.07) 




Near perfect 


0.96 (0.78-1.00) 


0.94 (0.71-1.00) 


0.96 (0.78-1.00) 


Smoker 


90.7 


0.72 


(0.13) 




Substantial 


0.64 (0.32-0.88) 


1.00 (0.87-1.00) 


1.00 (0.56-1.00) 


CVD 


90.7 


0.81 


(0.09) 




Near perfect 


0.86 (0.63-0.96) 


0.95 (0.75-1.00) 


0.95 (0.72-1.00) 


CKD 


81.4 


0.49 


(0.15) 




Moderate 


0.46 (0.20-0.74) 


0.97 (0.81-1.00) 


0.86 (0.42-0.99) 


ESRF 


1.00 


1.00 


(0.01) 




Near perfect 


1.00 (0.05-1.00) 


1.00 (0.90-1.00) 


1.00 (0.05-1.00) 


Other (non-listed) 


57.1 


0.09 


(0.12) 




Slight 


0.57 (0.40-0.73) 


0.57 (0.20-0.88) 


0.87 (0.65-0.97) 


High-risk history 


















Neuropathy 


93.0 


0.73 


(0.15) 




Substantial 


0.95 (0.80-0.99) 


0.83 (0.36-0.99) 


0.97 (0.84-1.00) 


PAD 


88.4 


0.77 


(0.10) 




Substantial 


0.86 (0.63-0.96) 


0.91 (0.69-0.98) 


0.90 (0.67-0.98) 


Previous foot ulcer 


79.1 


0.28 


(0.18) 




Fair 


0.84 (0.67-0.93) 


0.50 (0.14-0.86) 


0.91 (0.75-0.98) 


Current foot ulcer 


95.3 










0.98 (0.86-1.00) 


0 (0 - 0.95) 


0.98 (0.86-1.00) 


New foot ulcer 


93.0 


" 








0.95 (0.82-0.99) 


0 (0 - 0.95) 


0.98 (0.86-1.00) 


Previous amputation 


97.7 


0.95 


(0.05) 




Near perfect 


0.94 (0.69-1.00) 


1.00 (0.84-1.00) 


1.00 (0.76-1.00) 


Clinical diagnoses of high-risk factors 


















Neuropathy 


97.7 


0.90 


(0.10) 




Near perfect 


0.97 (0.85-1.00) 


1.00 (0.46-1.00) 


1.00 (0.88-1.00) 


Any PAD 


93.0 


0.86 


(0.8) 




Near perfect 


0.91 (0.69-0.98) 


0.95 (0.74-1.00) 


0.95 (0.74-1.00) 


Acute Charcot 


100 


1.00 


(0.01) 




Near perfect 


1.00 (0.05-1.00) 


1.00 (0.90-1.00) 


1.00 (0.05-1.00) 


Foot deformity 


86.0 


0.51 


(0.16) 




Moderate 


1.00 (0.87-1.00) 


0.4 (0.14-0.73) 


0.85 (0.69-0.94) 


Acute risk classification 


97.7 










1.00 (0.90-1.00) 


0 (0 - 0.95) 


0.98 (0.86-1.00) 


Clinical diagnoses of foot disease 


















Ulcer type: neuropathic (neuropathic or neuroisch.) 


90.7 


0.76 


(0.11) 




Substantial 


0.88 (0.72-0.96) 


1.0 (0.63-1.00) 


1.00 (0.86-1.00) 


Ulcer type: ischaemic 


79.1 


0.58 


(0.13) 




Moderate 


0.75 (0.51-0.90) 


0.83 (0.60-0.94) 


0.79 (0.54-0.93) 


(Neuroisch. or ischaemic) 


















Combined surface area mm 2 # 




0.93 


(0.87-0.96) 


Strong 








Any clinical infection 


88.4 


0.72 


(0.12) 




Substantial 


0.71 (0.42-0.90) 


0.97 (0.80-1.00) 


0.91 (0.57-1.00) 
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Table 4 Criterion validity summary statistics for senior clinician (Sensitivity, specificity and positive predictive value) & 
inter-rater reliability between expert and senior clinicians (Kappa and strength of agreement) (Continued) 



UTWCS (Grade 1A) 


83.7 


0.65 (12) 


Substantial 


0.85 (0.56-0.97) 


0.83 (0.64-0.93) 


0.71 (0.44-0.89) 


Deep ulcer 


95.3 


0.81 (0.13) 


Near perfect 


0.71 (0.30-0.95) 


0.97 (0.84-1.00) 


0.83 (0.36-0.99) 


(Grade 2 or 3)a 














Clinical management principles performed 














Ulcer debridement is required 


95.3 


0.66 (0.01) 


Substantial 


1.00 (0.89-1.00) 


1.00 (0.20-1.00) 


1.00 (0.89-1.00) 


Dressing is optimum 


100 






1.00 (0.90-1.00) 




1.00 (0.90-1.00) 


Antibiotics are required 


93.0 


0.82 (0.10) 


Near perfect 


0.83 (0.51-0.97) 


0.94 (0.77-0.99) 


0.83 (0.51-0.97) 


Off-loading is optimum 


93.0 


0.80 (0.11) 


Substantial 


0.80 (0.44-0.96) 


0.97 (0.82-1.00) 


0.89 (0.51-0.99) 


Footwear is optimum 


72.1 


0.31 (0.16) 


Fair 


0.50 (0.20-0.80) 


0.82 (0.64-0.92) 


0.45 (0.18-0.75) 


Educated patient 


97.7 






0.98 (0.86-1.00) 




1.00 (0.90-1.00) 



K = Kappa; (SE) = Standard Error; % = Percentage agreement; PPV = Positive Predictive Value. 

- = Measure of agreement was unable to be calculated because at least one variable was a constant. 

GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; PAD = Peripheral Arterial Disease; UTWCS = University of Texas Wound 
Classification System. 

# = Intraclass correlation (ICC) (Standard Deviation (SD)} for continuous data. 
A = Depth was determined from the UTWCS score. 



i.e. 'facility', 'visit type', 'todays visit to HRFS', 'separ- 
ation status', and 'date of referral'. Table 2 displays the 
general demographic, medical co-morbidity history, 
high-risk factor history, clinical diagnoses of high-risk 
factors and foot disease complications variable preva- 
lence for the three patient cohorts used. No significant 
differences were noted for these variables, except for 
diabetes duration and any other co-morbidity (p < 0.05). 
All cohorts contained variables with moderate prevalence 
rates (> 15%) [31], except for different health professionals 
attending previously, co-morbidity of end stage renal fail- 
ure (ESRF), acute Charcot and ischaemic ulcers. 

Criterion measure 

Table 3 displays the reliability results for the criterion 
measure of expert clinicians. Thirty-two items were able 
to be statistically tested. Nine items (28%) recorded near 
perfect categories of agreement, nine (28%) substantial/ 
strong categories, eleven (34%) moderate categories and 
three (9%) weak/fair categories. Thus, overall 29 (91%) of 
these items recorded at least a moderate category of reli- 
ability (K > 0.4; ICC > 0.75). The items recording weaker 
categories of reliability included other (non-listed) condi- 
tion, University of Texas Wound Classification System 
(UTWCS) grade and optimum footwear. 



moderate categories (> 0.7) for sensitivity, specificity and 
PPV respectively. Conversely, three (8%), five (14%) and 
four (11%) out of the 36 items recorded weak categories 
(< 0.7) for sensitivity, specificity and PPV respectively on 
both clinicians tested. The items registering weak categor- 
ies of validity (sensitivity, specificity or PPV on both clini- 
cians) included identifying a patient that had attended a 
GP, physician or surgeon; chronic kidney disease (CKD) or 
other (non-listed) condition; previous, current and new 
foot ulcer all had poor specificity in particular; and 
optimum footwear had poor PPVs. 

Inter-rater reliability 

Tables 4, 5 and 6 display the inter-rater reliability results 
for all three tests. Ninety-six different tests were able to 
be statistically tested for inter-rater reliability across the 
three different pairs of clinician agreement. Twenty four 
items (25%) recorded near perfect categories of agree- 
ment, 43 (45%) substantial/strong categories, 17 (18%) 
moderate categories and 12 (12%) weak/fair categories. 
Thus, overall 84 (88%) of these items recorded at least 
moderate categories of reliability (K > 0.4; ICC > 0.75). 
The items recording weak categories of reliability across 
two or more agreements tested included other (non-listed) 
condition, previous foot ulcer and optimum footwear. 



Criterion validity 

Tables 4 and 5 display the criterion validity results for 
both the senior (level 4) clinician and clinician (level 3) 
respectively tested against the criterion measure for each 
item. Thirty-six items were able to be statistically tested 
for sensitivity, specificity or positive predictive values 
(PPV) on both clinicians. Thus, 72 different tests were 
each performed for sensitivity, specificity and PPV. Sixty- 
one (85%), 59 (82%) and 60 (83%) items recorded at least 



Intra-rater reliability 

Table 6 also displays the intra-rater reliability results for 
the level 3 clinician. The median (interquartile range) 
period between the first and second ratings for the intra- 
rater reliability testing was 2(1-2) weeks. Twenty-three 
items were able to be statistically tested. Six items (26%) 
recorded near perfect categories of agreement, ten (43%) 
substantial/strong categories, four (17%) moderate cat- 
egories and three (13%) weak/fair categories. Thus, overall 
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Table 5 Criterion validity summary statistics for clinician (Sensitivity, specificity and positive predictive value) & 



inter-rater reliability between expert and clinician (Kappa and strength of agreement) 


QHRFF field 


% 


MSE) 


Strength of agreement [37] 


Sensitivity (95% CI) 


Specificity (95% CI) 


PPV (95% CI) 


General demographics 














Any indigenous status 


100 




- 


- 


1.00 (0.90-1.00) 


- 


Health professionals attending (in the past week) 










Podiatrist 


88.4 


- 


- 


0.88 (0.74-0.96) 


- 


0.88 (0.74-0.96) 


GP 


81.4 


0.39 (0.17) 


Fair 


0.40 (0.14-0.73) 


0.94 (0.78-0.99) 


0.67 (0.24-0.94) 


Nurse 


79.1 


0.37 (0.16) 


Fair 


0.33 (0.11-0.65) 


0.97 (0.81-1.00) 


0.80 (0.30-0.99) 


Orthotist 


100 


- 


- 


- 


1.00 (0.90-1.00) 


- 


Physician 


97.6 


0.66 (0.32) 


Substantia 


1.00 (0.05-1.00) 


0.98 (0.86-1.00) 


0.50 (0.03-0.97) 


Surgeon 


90.7 


0.45 (0.23) 


Moderate 


0.67 (0.13-0.98) 


0.93 (0.79-0.98) 


0.40 (0.07-0.83) 


Other 


100 


- 


- 


- 


0.98 (0.86-1.00) 


- 


Medical co morbidity history 














Any medical (diabetes) history 


100 


100 (0.01) 


Near perfect 


1.00 (0.88-1.00) 


1.00 (0.60-1.00) 


1.00 (0.88-1.00) 


Diabetes duration* 




0.95 (0.90-0.97) 


Strong 








Recent BGLs > 15 mmol/L 


90.7 


0.62 (0.17) 


Substantia 


1.00 (0.40-1.00) 


0.90 (0.75-0.97) 


0.50 (0.17-0.83) 


HbA1c result* 




1.00 (0.99-1.00) 


Strong 








Hypertension 


86.0 


0.68 (0.12) 


Substantial 


0.87 (0.68-0.96) 


0.85 (0.54-0.97) 


0.93 (0.75-0.99) 


Dyslipidaemia 


79.1 


0.58 (0.12) 


Moderate 


0.76 (0.54-0.90) 


0.83 (0.58-0.96) 


0.86 (0.64-0.96) 


Smoker 


88.4 


0.69 (0.13) 


Substantia 


0.73 (0.39-0.93) 


0.94 (0.78-0.99) 


0.80 (0.44-0.96) 


CVD 


79.1 


0.58 (0.12) 


Moderate 


0.71 (0.48-0.88) 


0.86 (0.64-0.96) 


0.83 (0.58-0.96) 


CKD 


86.0 


0.65 (0.13) 


Substantia 


0.69 (0.39-0.90) 


0.93 (0.76-0.99) 


0.82 (0.48-0.97) 


ESRF 


97.7 


0.66 (0.32) 


Substantial 


1 .00 (0.05-1 .00) 


0.98 (0.86-1.00) 


0.50 (0.03-0.97) 


Other (non-listed) 


79.1 


0.19 (0.19) 


Slight 


0.89 (0.73-0.96) 


0.29 (0.05-0.7) 


0.86 (0.70-0.95) 


High-risk history 














Neuropathy 


93.0 


0.73 (0.15) 


Substantia 


0.95 (0.80-0.99) 


0.83 (0.36-0.99) 


0.97 (0.84-1.00) 


PAD 


86.0 


0.72 (0.10) 


Substantia 


0.76 (0.52-0.91) 


0.95 (0.75-1.00) 


0.94 (0.69-1.00) 


Previous foot ulcer 


79.1 


0.28 (0.18) 


Fair 


0.84 (0.67-0.93) 


0.50 (0.14-0.86) 


0.91 (0.75-0.98) 


Current foot ulcer 


97.7 




- 


1.00 (0.90-1.00) 


0 (0 - 0.95) 


0.98 (0.86-1.00) 


New foot ulcer 


97.4 




- 


0.90 (0.76-0.97) 


0 (0 - 0.95) 


0.97 (0.85-0.99) 


Previous amputation 


95.3 


0.90 (0.07) 


Near perfect 


0.88 (0.62-0.98) 


1.00 (0.84-1.00) 


1.00 (0.75-1.00) 


Clinical diagnoses of high-risk factors 










Neuropathy 


100 


1.00 (0.01) 


Near perfect 


1.00 (0.89-1.00) 


1.00 (0.46-1.00) 


1.00 (0.89-1.00) 


Any PAD 


95.3 


0.91 (0.06) 


Near perfect 


0.91 (0.69-0.98) 


1.00 (0.81-1.00) 


1.00 (0.80-1.00) 


Acute Charcot 


100 


1.00 (0.01) 


Near perfect 


1.00 (0.05-1.00) 


1.00 (0.90-1.00) 


1.00 (0.05-1.00) 


Foot deformity 


93.0 


0.80 (0.11) 


Substantial 


0.97 (0.82-1 .00) 


0.80 (0.44-0.96) 


0.94 (0.79-0.99) 


Acute risk classify 


97.7 






1.00 (0.90-1.00) 


0 (0 - 0.95) 


0.98 (0.86-1.00) 


Clinical diagnoses of foot disease 












Ulcer type: neuropathic 


83.7 


0.55 (0.14) 


Moderate 


1.00 (0.86-1.00) 


0.46 (0.20-0.74) 


0.81 (0.64-0.91) 


(Neuropathic or neuroisch.) 














Ulcer type is ischaemic 


79.1 


0.57 (0.13) 


Moderate 


0.74 (0.48-0.90) 


0.83 (0.62-0.95) 


0.78 (0.52-0.93) 


(Neuroisch. or ischaemic) 














Combined surface area mm 2 # 




0.99 (0.99-1.00) 


Strong 








Any clinical infection 


81.4 


0.58 (0.13) 


Moderate 


0.71 (0.42-0.90) 


0.86 (0.67-0.95) 


0.71 (0.42-0.90) 
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Table 5 Criterion validity summary statistics for clinician (Sensitivity, specificity and positive predictive value) & 
inter-rater reliability between expert and clinician (Kappa and strength of agreement) (Continued) 



UTWCS (Grade 1A) 


81.4 


0.61 (0.12) 


Substantial 


0.86 (0.56-0.97) 


0.79 (0.60-0.91) 


0.69 (0.41-0.86) 


Deep ulcer 


93.0 


0.73 (0.15) 


Substantial 


0.71 (0.30-0.95) 


0.97 (0.84-1.00) 


0.83 (0.36-0.99) 


(Grade 2 or 3)a 














Clinical management principles performed 










Ulcer debridement is required 


95.3 


0.39 (0.22) 


Fair 


1.00 (0.89-1.00) 


0.50 (0.03-0.97) 


0.98 (0.86-1.00) 


Dressing is optimum 


97.7 






0.98 (0.86-1.00) 




1.00 (0.90-1.00) 


Antibiotics are required 


90.7 


0.76 (0.11) 


Substantial 


0.75 (0.43-0.93) 


0.97 (0.81-1.00) 


0.90 (0.54-0.99) 


Off-loading is optimum 


79.1 


0.50 (0.14) 


Moderate 


0.80 (0.44-0.96) 


0.79 (0.60-0.90) 


0.53 (0.27-0.78) 


Footwear is optimum 


79.1 


0.47 (0.15) 


Moderate 


0.70 (0.35-0.92) 


0.82 (0.64-0.92) 


0.54 (0.26-0.80) 


Educated patient 


100 






1.00 (0.90-1.00) 




1.00 (0.90-1.00) 



University of Texas Wound 



K = Kappa; (SE) = Standard Error; % = Percentage agreement; PPV = Positive Predictive Value. 

- = Measure of agreement was unable to be calculated because at least one variable was a constant. 

GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; PAD = Peripheral Arterial Disease; UTWCS = 

Classification System. 

*= Weighted Kappa used for ordinal data. 

# = Intraclass correlation (ICC) (Standard Deviation (SD)} for continuous data. 
A = Depth was determined from the UTWCS score. 



20 (87%) of these items recorded at least a moderate cat- 
egory of reliability (K > 0.4; ICC > 0.75). Those items scor- 
ing weak categories of agreement were foot deformity, 
optimal offloading and optimal footwear. 

Discussion 

The QHHRF appears to be the first multi-item tool 
developed and tested to identify multiple high-risk factors 
and foot disease complications in multiple at risk popula- 
tions. Our findings indicate that the majority of the tool's 
items demonstrates at least moderate categories of validity 
(face, content and criterion validity) and reliability (inter- 
rater and intra-rater); particularly those in the domains of 
identifying relevant medical co-morbidity history, and 
clinical diagnoses of high-risk factors and foot disease 
complications. However, some items in the domains of 
identifying different health professionals previously at- 
tending the patient and general clinical management 
principles performed appear to have weaker categories 
of validity and reliability that need addressing in future 
versions of the tool. 

The QHRFF went through a number of developmental, 
validity and reliability tests to determine its status as a 
valid and reliable tool. The magnitude of validity (or ac- 
curacy) in this study was evaluated via the methods of 
face, content and importantly criterion validity. Face 
and content validity are considered the least robust of 
the validity tests due to their inherent subjectivity [31,33]. 
However, they are considered important factors in the de- 
velopment phase to ensure the tool can actually measure 
the general identified construct [31,33]; in this case foot 
disease. At the completion of phase one, the expert panel 
and network stakeholders endorsements implied high 
practicality, face and content validity. 



Criterion validity is considered to be the most objective 
validity test [31,33]. To test concurrent criterion validity, 
as used by this study, a satisfactory criterion measure must 
be used. In this study the criterion measure to test criter- 
ion validity was an expert's clinical diagnosis. Overall, the 
'expert' criterion measure used for this study were rated as 
having at least moderate categories of reliability for all but 
three items; any other (non-listed) co-morbidity, optimum 
footwear and UTWCS grade. The other (non-listed) co- 
morbidity and optimum footwear items rated in the weak 
categories in most other inter- or intra-rater reliability 
agreement tested and should be reviewed or removed in 
future versions of the QHRFF. The weak-moderate agree- 
ment for the UTWCS grade was unexpected as this tool 
has been validated extensively in the past [62] , however, all 
other inter- or intra-rater agreements on the UTWCS 
were rated as substantial/ strong categories of reliability. 
Thus, it is recommended that the UTWCS grade be 
retained in future with more emphasis applied on the 
UTWCS grading system in the training sessions provided. 
It is certainly possible that the criterion measure for each 
item may have been more robust if the individual gold 
standard test for each individual item were used; for ex- 
ample an angiogram to diagnose peripheral arterial disease 
[22] or nerve conduction studies to diagnose neuropathy 
[44] . However, such an approach would have been particu- 
larly resource and time intensive and was thus not prac- 
tical within the resources available to this study. 

The QHRFF items were tested for concurrent criterion 
validity by using two different blinded representative gen- 
eral clinicians' ratings compared to the criterion measure 
(a reliable expert's diagnosis) on the same patients. Positive 
predictive values from this testing indicated the majority 
of QHRFF items had at least moderate validity when used 
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Table 6 Reliability measure of agreement summary statistics: Inter-rater (Senior clinician & clinician) & intra-rater 
(Clinician) 

Inter-rater reliability Intra-rater reliability 

QHRFF field % K (SE) Strength of agreement [37] % K (SE) Strength of agreement [37] 

General demographics 

Indigenous status* 83.7 - - 100 1.00 (0.01) Near Perfect 

Health professionals attending (in the past week) 



Podiatrist 


88.4 


- 


- 


N/A 


N/A 


N/A 


GP 


81.4 


0.45 (0.15) 


Moderate 


N/A 


N/A 


N/A 


Nurse 


81.4 


0.47 (0.14) 


Moderate 


N/A 


N/A 


N/A 


Orthotist 


100 


- 


- 


N/A 


N/A 


N/A 


Physician 


90.7 


- 


- 


N/A 


N/A 


N/A 


Surgeon 


90.7 


0.45 (0.23) 


Moderate 


N/A 


N/A 


N/A 


Other 


97.7 


- 


- 


N/A 


N/A 


N/A 


Medical co morbidity history 














Medical (diabetes) history 


95.3 


0.94 (0.05) 


Near perfect 


100 


1.00 (0.01) 


Near Perfect 


Diabetes duration* 




0.94 (0.87-0.97) 


Strong 




1.00 (0.99-1.00) 


Strong 


Recent BGLs >15 mmol/L* 


83.7 


0.67 (0.12) 


Substantial 


N/A 


N/A 


N/A 


HbA1c result* 




1.00 (1.00-1.00) 


Strong 




1.00 (1.00-1.00) 




Hypertension 


88.4 


0.73 (0.11) 


Substantial 


89.5 


0.78 (0.15) 


Substantial 


Dyslipidaemia 


83.7 


0.67 (0.11) 


Substantial 


94.7 


0.90 (0.10) 


Near Perfect 


Smoker 


88.4 


0.64 (0.15) 


Substantial 


94.7 


0.83 (0.17) 


Near Perfect 


CVD 


88.4 


0.76 (0.10) 


Substantial 


84.2 


0.69 (0.17) 


Substantial 


CKD 


86.1 


0.58 (0.15) 


Moderate 


94.7 


0.86 (0.14) 


Near Perfect 


ESRF 


97.7 


0.66 (0.32) 


Substantial 


100 


- 


- 


Other (non-listed) 


69.1 


0.34 (0.11) 


Fair 


78.9 


0.57 (0.19) 


Moderate 


High-risk history 














Neuropathy 


90.7 


0.66 (0.16) 


Substantial 


94.7 


- 


- 


PAD 


83.7 


0.67 (0.11) 


Substantial 


78.9 


0.60 (0.16) 


Moderate 


Previous foot ulcer 


90.7 


0.72 (0.13) 


Substantial 


94.7 


0.64 (0.33) 


Substantial 


Current foot ulcer 


97.7 


- 


- 


100 


- 


- 


New foot ulcer 


97.4 


- 


- 


100 


- 


- 


Previous amputation 


97.7 


0.95 (0.05) 


Near perfect 


100 


1.00 (0.01) 


Near Perfect 


Clinical diagnoses of high-risk factors 












Neuropathy 


97.7 


0.90 (0.10) 


Near perfect 


94.7 






PAD* 


97.9 


0.97 (0.03) 


Near perfect 


84.2 


0.69 (0.16) 


Substantial 


Acute Charcot 


100 


1.00 (0.01) 


Near perfect 


100 






Foot deformity 


83.7 


0.38 (0.18) 


Fair 


84.2 


0.31 (0.30) 


Fair 


Risk classification 


100 






100 






Clinical diagnoses of foot disease 












Ulcer type* 


88.4 


0.81 (0.08) 


Near perfect 


84.2 


0.77 (0.14) 


Substantial 


Combined surface area mm 2 # 




0.94 (0.89-0.97) 


Strong 




1.00 (1.00-1.00) 


Strong 


Clinical signs infection* 


76.7 


0.60 (0.12) 


Moderate 


89.5 


0.46 (0.32) 


Moderate 


UTWCS grade* 




0.90 (0.83-0.95) 


Strong 




0.98 (0.94-0.99) 


Strong 


Ulcer depths 


95.3 


0.90 (0.10) 


Near perfect 


94.7 


0.79 (0.23) 


Substantial 
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Table 6 Reliability measure of agreement summary statistics: Inter-rater (Senior clinician & clinician) & intra-rater 
(Clinician) (Continued) 



Clinical management principles performed 

Debrided ulcer/callus* 97.7 0.66 (0.32) 

Dressing optimum* 97.7 

Antibiotics required* 79.1 0.54(0.12) 

Off-loading optimum* 72.1 0.36(0.13) 

Footwear optimum* 67.4 0.33(0.14) 

Educated patient* 97.7 



Substantial 

Moderate 

Fair 

Fair 



94.7 
100 
89.5 
68.4 
68.4 
89.5 



0.44 (0.33) 
0.32 (0.23) 
0.37 (0.21) 



Moderate 

Fair 

Fair 



K = Kappa; (SE) = Standard Error; - = Measure of agreement was unable to be calculated because at least one variable was a constant. 

GP = General Practitioner; CVD = Cardiovascular Disease; ESRF = End Stage Renal Failure; N/A = Not applicable to test due to unstable rating environment; PAD = 
Peripheral Arterial Disease; UTWCS = University of Texas Wound Classification System. 
*= Weighted Kappa used for ordinal data; 

# = Intraclass correlation (ICC) (Standard Deviation (SD)) for continuous data. 



: Depth was determined from the UTWCS score. 



by clinicians with different levels of expertise. The high 
positive predictive values suggest that the proportion of 
people with a positive test result for individual QHRFF 
items actually had the relevant medical co-morbidity, 
high-risk factor or foot disease complication of inter- 
est. Although there were a few notable exceptions such 
as identifying different health professionals attending 
previously, recent blood glucose levels (BGLs) > 15 mmol/L, 
ESRF, optimal offloading and optimum footwear. How- 
ever, overall the QHRFF tool appears to demonstrate 
acceptable validity in the majority of its items to be con- 
sidered a valid tool to test the foot disease construct in 
multiple at risk populations. 

Reliability (or consistency) is a major prerequisite of any 
useful items to be measured [31,33]. In this study inter- 
rater and intra-rater measures of agreement were used for 
reliability. Inter-rater reliability was primarily assessed 
using three clinicians with different representative levels 
of expertise in managing foot disease. Overall, all items 
consistently displayed at least moderate categories of reli- 
ability, except for any other (non-listed) co-morbidity, 
optimum footwear and previous foot ulcer. Optimum 
footwear was also identified to have weak categories of 
intra-rater reliability; along with identifying foot deformity 
and optimum offloading. Thus, the QHRFF tool appears 
to demonstrate satisfactory reliability to collect the ma- 
jority of items in the foot disease construct. However, 
the authors recognise that it would have been preferable 
to test reliability with more clinicians across a larger 
sample of patients, yet this was beyond the resourcing 
available for this study. 

Notwithstanding, the aforementioned methodological 
limitations, the study does incorporate many best practice 
research design methods for testing validity and reliability. 
These methods included testing the reliability of the cri- 
terion measure to diagnose and measure each item, testing 
the subsequent concurrent criterion validity of representa- 
tive clinicians, and testing the reliability of the tool when 



used by clinicians with different levels of expertise. 
Furthermore, the study employed patient samples that 
were highly representative of the demographics and 
co-morbidity prevalence rates found in multiple 'at 
risk' populations for high risk factors and foot disease 
complications. These high prevalence rates not only 
suggest the tool was tested in the construct it was de- 
signed to measure, but literature suggests higher preva- 
lence rates improve the statistical robustness of validity 
and reliability results [31]. Lastly, many existing validated 
single-item tools [23,46,60,62] were incorporated within 
the QHRFF tool and this potentially adds weight to the 
validity reported in this study. Thus, overall the QHRFF 
tool appears to be a valid and reliable tool to collect the 
vast majority of items contained within the foot disease 
construct and can be used as a general tool to comple- 
ment existing specific tools in the field of foot disease. 

Limitations 

Several limitations have already been outlined in this 
study including the use of a general criterion measure of 
an experts' clinical diagnoses for each item, only using a 
limited number of representative clinicians and testing 
intra-rater reliability on one clinician in a small sample. 
Other limitations include: not performing a systematic 
literature review (and thus some evidence based items 
and tools may have been overlooked); using only podia- 
trists as the clinical raters; using historically defined 
strength categories for validity and reliability; not testing 
the tool for construct validity, predictive validity, or fac- 
tor analysis; and the settings used were existing HRFS 
only. It is recommended that any future research into 
this tool should address these methodological limitations 
by investigating the factor analysis, construct and pre- 
dictive criterion validity of the tool's items in a much 
larger and broader sample of patients with more multi- 
disciplinary clinician raters. Furthermore, if resources 
permit, the use of individual gold standard criterion 
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measures and/or existing concurrent validated single- 
item tools should be considered. 

However, there are a number of recommendations from 
this study's findings that should be considered for imple- 
mentation in future versions of the QHRFF tool. Firsdy, 
those items reporting weaker categories of validity or reli- 
ability and that potentially collect duplicate information to 
other more reliable items should be removed; including 
any other (non-listed) co-morbidity, combined surface 
area (change), recent BGLs greater than 15 mmol/L and 
new ulcer. Secondly, some items' definitions should be 
reviewed in an attempt to improve the item's future reli- 
ability and validity. For example, previous foot ulcer 
definition could be modified to identify a previously 
"healed" foot ulcer, 'health professionals attending' pre- 
viously requires an exact retrospective time period that 
patients were in attendance in alignment with similar 
literature (such as "in the previous two weeks") [64,65] 
and optimum offloading should have a more explicit 
definition referring directly to non-removable offload- 
ing definitions in the existing literature [14]. Thirdly, 
the validation of criteria for identifying a foot deformity 
and optimum footwear are urgently required for these 
'at risk' populations. To the best of the author's knowledge 
such tools for these items have been developed [14,50] yet 
are to be validated and this study was no exception. How- 
ever, in the meantime the authors recommend adding the 
numerical foot deformity score to the QHRFF to improve 
accuracy and test for validity. Lastly, a systematic literature 
review for other reported independent associates for 
foot disease (such as cancer, arthritis, depression, trauma, 
vision impairment, mobility impairment and social deter- 
minant factors) and other outcome measures for foot dis- 
ease (such as quality of life measures) should be performed 
to ensure all 'at risk' populations and outcome measures 
for foot disease are identified, considered and potentially 
included in the next QHRFF version. 

Conclusions 

The QHRFF tool appears to be the first multi-item tool 
developed and tested to identify multiple high-risk factors 
and foot disease complications in multiple at risk popu- 
lations. Acceptable validity and reliability were demon- 
strated for the majority of items; particularly in the 
domains of identifying relevant medical co-morbidity 
history and clinically diagnosing high-risk factors and 
foot disease complications. However, recommendations 
to remove, add or redefine items registering weak validity 
or reliability scores should be implemented to improve 
future versions of the QHRFF tool; particularly in the 
domains of health professionals attending and clinical 
management principles. Overall, the QHRFF tool ap- 
pears to demonstrate the practicality, validity and reli- 
ability required to facilitate robust clinical assessment 



and data capture to measure the large burden of foot 
disease facing our multiple at risk populations. 
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